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Abstract 

We present paired learning and inferenee 
algorithms for signifieantly redueing eom- 
putation and inereasing speed of the veetor 
dot produets in the elassifiers that are at the 
heart of many NLP eomponents. This is 
aeeomplished by partitioning the features 
into a sequenee of templates whieh are or¬ 
dered sueh that high eonfidenee ean of¬ 
ten be reaehed using only a small fraetion 
of all features. Parameter estimation is 
arranged to maximize aeeuraey and early 
eonfidenee in this sequenee. Our approaeh 
is simpler and better suited to NLP than 
other related easeade methods. We present 
experiments in left-to-right part-of-speeeh 
tagging, named entity reeognition, and 
transition-based dependeney parsing. On 
the typieal benehmarking datasets we ean 
preserve POS tagging aeeuraey above 97% 
and parsing LAS above 88.5% both with 
over a five-fold reduetion in run-time, and 
NER FI above 88 with more than 2x in- 
erease in speed. 

1 Introduction 

Many NLP tasks sueh as part-of-speeeh tagging, 
parsing and named entity reeognition have beeome 
suffieiently aeeurate that they are no longer solely 
an objeet of researeh, but are also widely deployed 
in produetion systems. These systems ean be run 
on billions of doeuments, making the effieieney 
of inferenee a signifieant eoneern—impaeting not 
only wall-eloek running time but also eomputer 
hardware budgets and the earbon footprint of data 
eenters. 

This paper deseribes a paired learning and infer¬ 
enee approaeh for signifieantly redueing eomputa- 
tion and inereasing speed while preserving aeeu- 
racy in the linear elassifiers typieally used in many 


NLP tasks. The heart of the predietion eomputa- 
tion in these models is a dot-produet between a 
dense parameter veetor and a sparse feature vee¬ 
tor. The bottleneek in these models is then often 
a eombination of feature extraetion and numeri- 
eal operations, eaeh of whieh seale linearly in the 
size of the feature veetor. Feature extraetion ean 
be even more expensive than the dot produets, in¬ 
volving, for example, walking sub-graphs, lexieon 
lookup, string eoneatenation and string hashing. 
We note, however, that in many eases not all of 
these features are neeessary for aeeurate predie¬ 
tion. For example, in part-of-speeeh tagging if we 
see the word “the,” there is no need to perform a 
large dot produet or many string operations; we 
ean aeeurately label the word a Determiner us¬ 
ing the word identity feature alone. In other eases 
two features are suffieient: when we see the word 
“hits” preeeded by a Cardinal (e.g. “two hits”) 
we ean be eonfident that it is a NOUN. 

We present a simple yet novel approaeh to im¬ 
prove proeessing speed by dynamieally determin¬ 
ing on a per-instanee basis how many features are 
neeessary for a high-eonfidenee predietion. Our 
features are divided into a set of feature templates, 
sueh as current-token or previous-tag in the ease of 
POS tagging. At training time, we determine an 
ordering on the templates sueh that we ean approx¬ 
imate model seores at test time by inerementally 
ealeulating the dot produet in template ordering. 
We then use a running eonfidenee estimate for the 
label predietion to determine how many terms of 
the sum to eompute for a given instanee, and pre¬ 
diet onee eonfidenee reaehes a eertain threshold. 

In similar work, easeades of inereasingly eom- 
plex and high-reeall models have been used for 
both struetured and unstruetured predietion. Viola 
and Jones (2001) use a easeade of boosted mod¬ 
els to perform faee deteetion. Weiss and Taskar 
(2010) add inereasingly higher-order dependen- 
eies to a graphieal model while filtering the out- 



put domain to maintain tractable inference. While 
most traditional cascades pass instances down to 
layers with increasingly higher recall, we use a 
single model and accumulate the scores from each 
additional template until a label is predicted with 
sufficient confidence, in a stagewise approxima¬ 
tion of the full model score. Our technique applies 
to any linear classifier-based model over fealure 
femplafes wifhouf changing fhe model sfrucfure or 
decreasing prediction speed. 

Mosf similarly fo our work, Weiss and Taskar 
(2013) improve performance for several sfrucfured 
vision fasks by dynamically selecting fealures af 
runfime. However, fhey use a reinforcemenl learn¬ 
ing approach whose compufafional Iradeoffs are 
beffer suifed fo vision problems wifh expensive 
fealures. Oblaining a speedup on fasks wifh com¬ 
paratively cheap fealures, such as parl-of-speech 
lagging or Iransilion-based parsing, requires an 
approach wifh less overhead. In fad, fhe mosf al- 
Iracfive aspecl of our approach is lhaf if speeds up 
melhods lhaf are already among Ihe faslesl in NLP. 

We apply our melhod lo lefl-lo-righl parl-of- 
speech lagging in which we achieve accuracy 
above 97% on Ihe Penn Treebank WSJ corpus 
while running more lhan live times faster lhan our 
97.2% baseline. We also achieve a live-fold in¬ 
crease in Iransilion-based dependency parsing on 
Ihe WSJ corpus while achieving an LAS jusl 1.5% 
lower lhan our 90.3% baseline. Named entity 
recognition also shows significanl speed increases. 
We furlher demonslrale lhal our melhod can be 
luned for 2.5 — 3.5x multiplicative speedups wilh 
nearly no loss in accuracy. 

2 Classification and Structured 
Prediction 

Our algorilhm speeds up prediction for multiclass 
classification problems where Ihe label sel can be 
Iraclably enumerated and scored, and Ihe per-class 
scores of inpul fealures decompose as a sum over 
multiple fealure templates. Frequenlly, classifica¬ 
tion problems in NLP are solved Ihrough Ihe use of 
linear classifiers, which compute scores for inpul- 
label pairs using a dol product These meel our ad¬ 
ditive scoring criteria, and our acceleration melh¬ 
ods are direclly applicable. 

However, in Ihis work we are interested 
in speeding up structured prediction problems, 
specifically parl-of-speech (POS) lagging and de¬ 
pendency parsing. We apply our classification 


algorilhms lo Ihese problems by reducing Ihem 
lo sequential prediction (Daume III el at, 2009). 
For POS lagging, we describe a sentence’s pari of 
speech annolalion by Ihe lefl-lo-righl sequence of 
lagging decisions for individual tokens (Gimenez 
and Marquez, 2004). Similarly, we implemenl our 
parser wilh a classifier lhal generates a sequence 
of shifl-reduce parsing Iransilions (Nivre, 2009). 

The use of sequential prediction to solve Ihese 
problems and olhers has a long history in prac¬ 
tice as well as Iheory. Seam (Daume III el at, 
2009) and DAgger (Ross el at, 2011) are Iwo pop¬ 
ular principled frameworks for reducing sequen¬ 
tial prediction to classification by learning a clas¬ 
sifier on additional synlhelic Iraining dala. How¬ 
ever, as we do in our experimenls, practitioners of¬ 
ten see good resulls by Iraining on Ihe gold stan¬ 
dard labels wilh an off-lhe-shelf classification al¬ 
gorilhm, as Ihough classifying IID data (Benglson 
and Rolh, 2008; Choi and Palmer, 2012). 

Classifier-based approaches to slruclured pre¬ 
diction are faster lhan dynamic programming 
since Ihey consider only a subsel of candidate oul- 
pul slruclures in a greedy manner. For exam¬ 
ple, Ihe Stanford CoreNLP classifier-based parl- 
of-speech tagger provides a 6.5x speed advantage 
over Iheir dynamic programming-based model, 
wilh lillle reduction in accuracy. Because our 
melhods are designed for Ihe greedy sequential 
prediction regime, we can provide furlher speed 
increases to Ihe faslesl inference melhods in NLP. 

3 Linear models 

Our base classifier for sequential prediction tasks 
will be a linear model. Given an inpul x G Tf, a sel 
of labels y, a fealure map <F(x, y), and a weighl 
vector w, a linear model predicls Ihe highesl- 
scoring label 

y* = argmax w • 4>(x,y). (1) 

y&y 

The parameter w is usually learned by minimizing 
a regularized {R) sum of loss functions {t} over Ihe 
Iraining examples indexed by i 

w* = argminN^l'(xi,yi, w) -|- R{'w). 

W 

I 

In Ihis paper, we partition Ihe fealures into a sel 
oi feature templates, so lhal Ihe weighls, fealure 
function, and dol producl factor as 

w • T>(x,y) = ^ Wj • T>j(x,y) (2) 
j 



for some set of feature templates {<I>j(x, y)}. 

Our goal is to approximate the dot produets in 
(1) suffieiently for purposes of predietion, while 
using as few terms of the sum in (2) as possible. 

4 Method 

We aeeomplish this goal by developing paired 
learning and inferenee proeedures for feature- 
templated elassifiers that optimize both aeeuraey 
and inferenee speed, using a proeess of dynamic 
feature selection. Sinee many deeisions are easy 
to make in the presenee of strongly predietive fea¬ 
tures, we would like our model to use fewer tem¬ 
plates when it is more eonfident. For a fixed, 
learned ordering of feature templates, we build up 
a veetor of class scores incrementally over each 
prefix of fhe sequence of femplafes, which we call 
fhe prefix scores. Once we reach a slopping crile- 
rion based on class confidence (margin), we slop 
compuling prefix scores, and predicl fhe currenl 
highesl scoring class. Our aim is lo Irain each pre¬ 
fix lo be as good a classifier as possible wilhoul 
fhe following femplafes, minimizing fhe number 
of femplafes needed for accurale predicfions. 

Given Ihis melhod for performing fasl inference 
on an ordered sel of fealure femplafes, if remains 
lo choose fhe ordering. In Section 4.5, we de¬ 
velop several melhods for picking lemplale order¬ 
ings, based on ideas from group sparsify (Yuan and 
Lin, 2006; Swirszcz el ah, 2009), and olher tech¬ 
niques for fealure subsel-seleclion (Kohavi and 
John, 1997). 

4.1 Definitions 

Given a model lhaf compufes scores addifively 
over lemplale-specific scoring functions as in (2), 
paramelers w, and an observalion x G Y, we can 
define fhe f’lh prefix score for label y ^ y as: 

i 

Pi^y{x,w) ^j{x,y), 

J=1 

or Pi^y when fhe choice of observalions and 
weighls is clear from conlexl. Abusing nolalion 
we also refer lo fhe veclor conlaining all i’lh prefix 
scores for observation x associated lo each label in 
y as Pi{x, w), or Pi when Ibis is unambiguous. 

Given a parameler m > 0, called fhe margin, 
we define a function h on prefix scores: 

h{Pi, y) = max{0, maxPjy — Pi^y + m} 
y'¥=y 


Algorithm 1 Inference 

Input: template parameters margin m 

and optional (for Irain time) Irue label y 
Initialize: i = 1 
while I > 0 Ai < k do 

I = maxy/ h{Pi, y') (fesl) or h{Pi, y) (Irain) 
f t— i -h 1 
end while 

return (Irain) or max^/ Pi^yi (fesl) 


Algorithm 2 Parameler Learning 

Input: examples {(xj, yi)}f, margin m 
Initialize: paramelers Wq = 0, f = 1 
while i < N do 

prefixes t— Infer(xj, Wj, m) 

Pi t— CompuleGradienl(prefixes) 
Wj+i t— UpdateParamefers(wi, yj) 
f ^ i -h 1 
end while 
return wat 


This is fhe familiar slruclured hinge loss func¬ 
tion as in sfruclured supporl veclor machines 
(Tsochanlaridis el ah, 2004), which has a mini¬ 
mum al 0 if and only if class y is ranked ahead of 
all olher classes by al leasl m. 

Using Ibis nolalion, fhe condilion lhaf some la¬ 
bel y be ranked firsl by a margin can be wril- 
len as h{Pi,y) = 0, and fhe condition lhaf any 
class be ranked firsl by a margin can be wriflen as 
maxy/ h{Pi, y') = 0. 

4.2 Inference 

As described in Algorilhm 1, al fesl lime we com- 
pule prefixes unlil some label is ranked ahead of 
all olher labels wilh a margin m, Ihen predicl wilh 
lhaf label. Al Irain time, we predicl until Ihe cor- 
recl label is ranked ahead wilh margin m, and re- 
lurn Ihe whole sel of prefixes for use by Ihe learn¬ 
ing algorilhm. If no prefix scores have a margin, 
Ihen we predicl wilh Ihe final prefix score involv¬ 
ing all Ihe fealure templates. 

4.3 Learning 

We splil learning into Iwo subproblems: firsl, 
given an ordered sequence of fealure templates 
and our inference procedure, we wish to learn pa¬ 
rameters lhal optimize accuracy while using as few 
of Ihose templates as possible. Second, given a 
melhod for Iraining fealure lemplaled classifiers. 



we want to learn an ordering of templates that op¬ 
timizes aeeuraey. 

We wish to optimize several different objee- 
tives during learning: template parameters should 
have strong predietive power on their own, but also 
work well when eombined with the seores from 
later templates. Additionally, we want to eneour- 
age well-ealibrated eonfidenee seores that allow us 
to stop predietion early without signifieant redue- 
tion in generalization ability. 

4.4 Learning the parameters 

To learn parameters that eneourage the use of few 
feature templates, we look at the model as out- 
putting not a single predietion but a sequenee of 
prefix predietions {Pi}. For eaeh training ex¬ 
ample, eaeh feature template reeeives a number 
of hinge-loss gradients equal to its distanee from 
the index where the margin requirement is finally 
reaehed. This is equivalenf fo freafing eaeh prefix 
as ifs own model for whieh we have a hinge loss 
funelion, and learning all models simulfaneously. 
Our high-level approaeh is deseribed in Algorifhm 
2 . 

Conerefely, for k fealure femplafes we opti¬ 
mize fhe following sfruefured max-margin objee- 
five (wifh fhe dependenee of P’s on w wriffen ex- 
plieifly where helpful): 

w* = arg min £{x, y, w) 

i^,y) 

A 

£(x,y,w) = ^h{Pi{x,y/),y) 

i=l 

i*y = min{i}f^i s.t. h{Pi,y) = 0 

The per-example gradienf of Ibis objeefive for 
weighfs Wj eorresponding fo fealure lemplale 
Ihen eorresponds fo 

d£ ^ 

3 

where we define 

y\oss{Pi,y) = arg max Pjy - m - I{y' = y), 
y' 

where I is an indiealor funelion of fhe label y, used 
fo define loss-augmenled inferenee. 

We add an £2 regularization lerm fo fhe objee- 
live, and lune fhe margin m and fhe regularizalion 
slrenglh fo Iradeoff belween speed and aeeuraey. 


In our experimenls, we used a developmenl sel fo 
ehoose a regularizer and margin lhal redueed lesl- 
lime speed as mueh as possible wilhouf deereasing 
aeeuraey. We Ihen varied fhe margin for lhal same 
model al lesl lime fo aehieve larger speed gains al 
fhe eosl of aeeuraey. In all experimenls, fhe mar¬ 
gin wifh which fhe model was Irained corresponds 
fo fhe largesl margin reporled, i.e. lhal wifh fhe 
highesl accuracy. 

4.5 Learning the template ordering 

We examine three approaches to learning the tem¬ 
plate ordering. 

4.5.1 Group Lasso and Group Orthogonal 
Matching Pursuit 

The Group Lasso regularizer (Yuan and Lin, 2006) 
penalizes the sum of £ 2 -norms of weights of fea¬ 
ture templates (different from what is commonly 
called “£ 2 ” regularization, penalizing squared £2 
norms), CiUmilb. where q is a weight for 
each template. This regularizer encourages entire 
groups of weights to be set to 0, whose templates 
can then be discarded from the model. By vary¬ 
ing the strength of the regularizer, we can learn an 
ordering of the importance of each template for a 
given model. The included groups for a given reg¬ 
ularization strength are nearly always subsets of 
one another (technical conditions for this to be true 
are given in Has tie et al. (2007)). The sequence 
of solutions for varied regularization strength is 
called the regularization path, and by slight abuse 
of terminology we use this to refer to the induced 
template ordering. 

An alternative and related approach to learn¬ 
ing template orderings is based on the Group Or¬ 
thogonal Matching Pursuit (GOMP) algorithm for 
generalized linear models (Swirszcz et ah, 2009; 
Lozano et ah, 2011), with a few modifications for 
the setting of high-dimensional, sparse NLP data 
(described in Appendix B). Orthogonal matching 
pursuit algorithms are a set of stagewise feature 
selection techniques similar to forward stagewise 
regression (Hastie et ah, 2007) and LARS (Efron 
et ah, 2004). At each stage, GOMP effectively 
uses each feature template to perform a linear re¬ 
gression to fit the gradient of the loss function. 
This attempts to find the correlation of each fea¬ 
ture subset with the residual of the model. It then 
adds the feature template that best tits this gradi¬ 
ent, and retrains the model. The main weakness of 
this method is that it fits the gradient of the training 



error which can rapidly overfit for sparse, high¬ 
dimensional data. Ultimately, we would prefer to 
use a development set for feature selection. 

4.5.2 Wrapper Method 

The wrapper method (Kohavi and John, 1997) 
is a meta-algorithm for feature selection, usually 
based on a validation set. We employ it in a stage- 
wise approach to learning a sequence of templates. 
Given an ordering of the initial sub-sequence and 
a learning procedure, we add each remaining tem¬ 
plate to our ordering and estimate parameters, se¬ 
lecting as the next template the one that gives the 
highest increase in development set performance. 
We begin the procedure with no templates, and re¬ 
peat the procedure until we have a total ordering 
over the set of feature templates. When learning 
the ordering we use the same hyperparameters as 
will be used during final training. 

While simpler than the Lasso and Matching 
Pursuit approaches, we empirically found this ap¬ 
proach to outperform the others, due to the neces¬ 
sity of using a development set to select features 
for our high-dimensional application areas. 

5 Related Work 

Our work is primarily inspired by previous re¬ 
search on cascades of classifiers; however, it dif¬ 
fers significantly by approximating the score of a 
single linear model—scoring as few of its features 
as possible to obtain sufficient confidence. 

We pose and address the question of whether a 
single, interacting set of parameters can be learned 
such that they efficiently both (1) provide high ac¬ 
curacy and (2) good confidence estimates through¬ 
out their use in the lengthening prefixes of the 
feature template sequence. (These two require¬ 
ments are both incorporated into our novel param¬ 
eter estimation algorithm.) In contrast, other work 
(Weiss and Taskar, 2013; He et ah, 2013) learns 
a separate classifier to determine when to add fea¬ 
tures. Such heavier-weight approaches are unsuit¬ 
able for our setting, where the core classifier’s fea¬ 
tures and scoring are already so cheap that adding 
complex decision-making would cause too much 
computational overhead. 

Other previous work on cascades uses a se¬ 
ries of increasingly complex models, such as the 
Viola-Jones face detection cascade of classifiers 
(2001), which applies boosted trees trained on 
subsets of features in increasing order of complex¬ 
ity as needed, aiming to reject many sub-image 


windows early in processing. We allow scores 
from each layer to directly affect the final predic¬ 
tion, avoiding duplicate incorporation of evidence. 

Our work is also related to the field of learn¬ 
ing and inference under test-time budget con¬ 
straints (Grubb and Bagnell, 2012; Trapeznikov 
and Saligrama, 2013). However, common ap¬ 
proaches to this problem also employ auxiliary 
models to rank which feature to add next, and 
are generally suited for problems where features 
are expensive to compute {e.g vision) and the ex¬ 
tra computation of an auxiliary pruning-decision 
model is offset by substantial reduction in fea¬ 
ture computations (Weiss and Taskar, 2013). Our 
method uses confidence scores directly from the 
model, and so requires no additional computation, 
making it suitable for speeding up classifier-based 
NLP methods that are already very fast and have 
relatively cheap features. 

Some cascaded approaches strive at each stage 
to prune the number of possible output structures 
under consideration, whereas in our case we fo¬ 
cus on pruning the input features. For example, 
Xu et al. (2013) learn a tree of classifiers that sub¬ 
divides the set of classes to minimize average test¬ 
time cost. Chen et al. (2012) similarly use a linear 
cascade instead of a tree. Weiss and Taskar (2010) 
prune output labels in the context of structured 
prediction through a cascade of increasingly com¬ 
plex models, and Rush and Petrov (2012) success¬ 
fully apply these structured prediction cascades to 
the task of graph-based dependency parsing. 

In the context of NLP, He et al. (2013) describe 
a method for dynamic feature template selection 
at test time in graph-based dependency parsing. 
Their technique is particular to the parsing task— 
making a binary decision about whether to lock in 
edges in the dependency graph at each stage, and 
enforcing parsing-specific, hard-coded constraints 
on valid subsequent edges. Furthermore, as de¬ 
scribed above, they employ an auxiliary model to 
select features. 

He and Eisner (2012) share our goal to speed 
test time prediction by dynamically selecting fea¬ 
tures, but they also learn an additional model on 
top of a fixed base model, rather than using the 
training objective of the model itself. 

While our comparisons above focus on other 
methods of dynamic feature selection, there also 
exists related work in the field of general (static) 
feature selection. The most relevant results come 



from the applications of group sparsity, such as 
the work of Martins et al. (2011) in Group Lasso 
for NLP problems. The Group Lasso regularizer 
(Yuan and Lin, 2006) sparsifies groups of feature 
weights (e.g. feature templates), and has been 
used to speed up test-time prediction by remov¬ 
ing entire templates from the model. The key dif¬ 
ference between this work and ours is that we se¬ 
lect our templates based on the test-time difficulty 
of the inference problem, while the Group Lasso 
must do so at train time. In Appendix A, we com¬ 
pare against Group Lasso and show improvements 
in accuracy and speed. 

Note that non-grouped approaches to selecting 
sparse feature subsets, such as boosting and £i reg¬ 
ularization, do not achieve our goal of fast test¬ 
time prediction in NLP models, as they would 
not zero-out entire templates, and still require the 
computation of a feature for every template for ev¬ 
ery test instance. 

6 Experimental Results 

We present experiments on three NLP tasks 
for which greedy sequence labeling has been 
a successful solution: part-of-speech tagging, 
transition-based dependency parsing and named 
entity recognition. In all cases our method 
achieves multiplicative speedups at test time with 
little loss in accuracy. 

6.1 Part-of-speech tagging 

We conduct our experiments on classifier-based 
greedy parf-of-speech fagging. Our baseline fag- 
ger uses fhe same feafures described in Choi and 
Palmer (2012). We evaluafe our models on fhe 
Penn Treebank WSJ corpus (Marcus el ah, 1993), 
employing fhe lypical splif of secfions used for 
parl-of-speech lagging: 0-18 Irain, 19-21 devel- 
opmenl, 22-24 lesl. The parameters of our mod¬ 
els are learned using AdaGrad (Duchi el ah, 2011) 
wilh £2 regularizalion via regularized dual averag¬ 
ing (Xiao, 2009), and we used random search on 
fhe developmenl sel fo selecl hyperparamelers. 

This baseline model (baseline) lags al a rate 
of approximately 23,000 tokens per second on a 
2010 2.1GHz AMD Opleron machine wilh ac¬ 
curacy comparable to similar laggers (Gimenez 
and Marquez, 2004; Choi and Palmer, 2012; 
Toulanova el ah, 2003). On Ihe same machine 
Ihe greedy Slanford CoreNLP lefl3words parl-of- 
speech lagger also lags al approximately 23,000 


Model/m 

Tok. 

Unk. 

Feat. 

Speed 

Baseline 

97.22 

88.63 

46 

lx 

Slagewise 

96.54 

83.63 

9.50 

2.74 

Fixed 

89.88 

56.25 

1 

16.16x 

Fixed 

94.66 

60.59 

3 

9.54x 

Fixed 

96.16 

87.09 

5 

7.02x 

Fixed 

96.88 

88.81 

10 

3.82x 

Dynamic/15 

96.09 

83.12 

1.92 

10.36X 

Dynamic/35 

97.02 

88.26 

4.33 

5.22x 

Dynamic/45 

97.16 

88.84 

5.87 

3.97x 

Dynamic/50 

97.21 

88.95 

6.89 

3.41x 


Table 1: Comparison of our models using differ- 
enl margins m, wilh speeds measured relative to 
Ihe baseline. We Irain a model as accurate as Ihe 
baseline while lagging 3.4x lokens/sec, and in an- 
olher model mainlain > 97% accuracy while lag¬ 
ging 5.2x, and > 96% accuracy wilh a speedup of 
10.3x. 

tokens per second. Significanlly higher absolute 
speeds for all melhods can be allained on more 
modern machines. 

We include additional baselines lhal divide Ihe 
fealures into templates, bul Irain Ihe templates’ pa¬ 
rameters more simply lhan our algorilhm. The 
stagewise baseline learns Ihe model parameters 
for each of Ihe templates in order, slarling wilh 
only one template—once each template has been 
Irained for a fixed number of iterations, lhal tem¬ 
plate’s parameters are fixed and we add Ihe nexl 
one. We also create a separalely-lrained baseline 
model for each fixed prefix of Ihe fealure templates 
(fixed). This shows lhal our speedups are nol sim¬ 
ply due to superfluous fealures in Ihe later tem¬ 
plates. 

Our main resulls are shown in Table 1. We in¬ 
crease Ihe speed of our baseline POS lagger by a 
factor of 5.2x wilhoul falling below 97% lesl ac¬ 
curacy. By luning our Iraining melhod to more 
aggressively prune templates, we achieve speed- 
ups of over lOx while providing accuracy higher 
lhan 96%. Il is worlh noting lhal Ihe resulls for 
our melhod (dynamic) are all oblained from a 
single Irained model (wilh hyperparamelers opti¬ 
mized for m = 50, which we observed gave a 
good speedup wilh nearly no lossin accuracy on 
Ihe developmenl sel), Ihe only difference being 
lhal we varied Ihe margin al lesl time. Superior 
resulls for m / 50 could likely be oblained by op- 





Figure 1: Left-hand plot depicts test accuracy as a function of the average number of templates used 
to predict. Right-hand plot shows speedup as a function of accuracy. Our model consistently achieves 
higher accuracy while using fewer templates resulting in the best ratio of speed to accuracy. 


timizing hyperparameters for the desired margin. 

Results show our method (dynamic) learns to 
dynamically select the number of templates, often 
using only a small fraction. The majority of test 
tokens can be tagged using only the first few tem¬ 
plates: just over 40% use one template, and 75% 
require at most four templates, while maintaining 
97.17% accuracy. On average 6.71 out of 46 tem¬ 
plates are used, though a small set of complicated 
instances never surpass the margin and use all 46 
templates. The right hand plot of Figure 1 shows 
speedup vs. accuracy for various settings of the 
confidence margin m. 

The left plot in Figure 1 depicts accuracy as a 
function of the number of templates used at test 
time. We present results for both varying the 
number of templates directly (dashed) and margin 
(solid). The baseline model trained on all tem¬ 
plates performs very poorly when using margin- 
based inference, since its training objective does 
not learn to predict with only prefixes. When pre- 
dicfing using a fixed subsef of templates, we use a 
different baseline model for each one of the 46 to¬ 
tal template prefixes, learned with only those fea¬ 
tures; we then compare the test accuracy of our 
dynamic model using template prefix i to the base¬ 
line model trained on the fixed prefix i. Our model 
performs just as well as these separately trained 
models, demonstrating that our objective learns 
weights that allow each prefix to act as its own 
high-quality classifier. 


6.1.1 Learning the template ordering 

As described in Section 4.5, we experimented on 
part-of-speech tagging with three different algo¬ 
rithms for learning an ordering of feature tem¬ 
plates: Group Lasso, Group Orthogonal Matching 
Pursuit (GOMP), and the wrapper method. For 
the case of Group Lasso, this corresponds to the 
experimental setup used when evaluating Group 
Lasso for NLP in Martins et al. (2011). As detailed 
in the part-of-speech tagging experiments of Ap¬ 
pendix A, we found the wrapper method to work 
best in our dynamic prediction setting. Therefore, 
we use it in our remaining experiments in pars¬ 
ing and named entity recognition. Essentially, the 
Group Lasso picks small templates too early in 
the ordering by penalizing template norms, and 
GOMP picks large templates too early by overfit¬ 
ting the train error. 

6.2 Transition-based dependency parsing 

We base our parsing experiments on the greedy, 
non-projective transition-based dependency parser 
described in Choi and Palmer (2011). Our model 
uses a total of 60 feature templates based mainly 
on the word form, POS tag, lemma and assigned 
head label of current and previous input and stack 
tokens, and parses about 300 sentences/second on 
a modest 2.1GHz AMD Opteron machine. 

We train our parser on the English Penn Tree- 
Bank, learning the parameters using AdaGrad and 
the parsing split, training on sections 2-21, testing 
on section 23 and using section 22 for develop¬ 
ment and the Stanford dependency framework (de 

























Figure 2: Parsing speedup as a function of accu¬ 
racy. Our model achieves the highest accuracy 
while using the fewest feature templates. 

Marneffe and Manning, 2008). POS tags were au¬ 
tomatically generated via 10-way jackknifing us¬ 
ing the baseline POS model described in the pre¬ 
vious section, trained with AdaGrad using £2 reg¬ 
ularization, with parameters tuned on the develop¬ 
ment set to achieve 97.22 accuracy on WSJ sec¬ 
tions 22-24. Lemmas were automatically gener¬ 
ated using the ClearNLP morphological analyzer. 
We measure accuracy using labeled and unlabeled 
attachment scores excluding punctuation, achiev¬ 
ing a labeled score of 90.31 and unlabeled score 
of 91.83, which are comparable to similar greedy 
parsers (Choi and Palmer, 2011; Honnibal and 
Goldberg, 2013). 

Our experimental setup is the same as for part- 
of-speech tagging. We compare our model (dy¬ 
namic) to both a single baseline model trained on 
all features, and a set of 60 models each trained 
on a prefix of feature templates. Our experiments 
vary the margin used during prediction (solid) as 
well as the number of templates used (dashed). 

As in part-of-speech tagging, we observe sig¬ 
nificant test-time speedups when applying our 
method of dynamic feature selection to depen¬ 
dency parsing. With a loss of only 0.04 labeled at¬ 
tachment score (LAS), our model produces parses 
2.7 times faster than the baseline. As listed in Ta¬ 
ble 2, with a more aggressive margin our model 
can parse more than 3 times faster while remain¬ 
ing above 90% LAS, and more than 5 times faster 
while maintaining accuracy above 88.5%. 

In Figure 2 we see not only that our dynamic 
model consistently achieves higher accuracy while 


Model/m 

LAS 

UAS 

Feat. 

Speed 

Baseline 

90.31 

91.83 

60 

lx 

Fixed 

65.99 

70.78 

1 

27.5x 

Fixed 

86.87 

88.81 

10 

5.51x 

Fixed 

88.76 

90.51 

20 

2.83x 

Fixed 

89.04 

90.71 

30 

L87x 

Dynamic/6.5 

88.63 

90.36 

7.81 

5.16x 

Dynamic/7.1 

89.07 

90.73 

8.57 

4.66x 

Dynamic/10 

90.16 

91.70 

13.27 

3.17X 

Dynamic/11 

90.27 

91.80 

15.83 

2.71x 


Table 2: Comparison of our baseline and tern- 
plated models using varying margins m and num¬ 
bers of templates. 

using fewer templates, but also that our model (dy¬ 
namic, dashed) performs exactly as well as sep¬ 
arate models trained on each prefix of templates 
(baseline, dashed), demonstrating again that our 
training objective is successful in learning a single 
model that can predict as well as possible using 
any prefix of feature templates while successfully 
selecting which of these prefixes to use on a per- 
example basis. 

6.3 Named entity recognition 

We implement a greedy left-to-right named entity 
recognizer based on Ratinov and Roth (2009) us¬ 
ing a total of 46 feature templates, including sur¬ 
face features such as lemma and capitalization, 
gazetteer look-ups, and each token’s extended pre¬ 
diction history, as described in (Ratinov and Roth, 
2009). Training, tuning, and evaluation are per¬ 
formed on the CoNLL 2003 English data set with 
the BILOU encoding to denote label spans. 

Our baseline model achieves FI scores of 88.35 
and 93.37 on the test and development sets, re¬ 
spectively, and tags at a rate of approximately 
5300 tokens per second on the hardware described 
in the experiments above. We achieve a 2.3x 
speedup while maintaining FI score above 88 on 
the test set. 

7 Conclusions and Future Work 

By learning to dynamically select the most predic¬ 
tive features at test time, our algorithm provides 
significant speed improvements to classifier-based 
structured prediction algorithms, which them¬ 
selves already comprise the fastest methods in 
NLR Further, these speed gains come at very lit- 















Model/m 

Test FI 

Feat. 

Speed 

Baseline 

88.35 

46 

lx 

Fixed 

65.05 

1 

19.08X 

Fixed 

85.00 

10 

2.14x 

Fixed 

85.81 

13 

1.87x 

Dynamie/3.0 

87.62 

7.23 

2.59x 

Dynamie/4.0 

88.20 

9.45 

2.32x 

Dynamie/5.0 

88.23 

12.96 

1.96x 


Table 3: Comparison of our baseline and tern- 
plated NER models using varying margin m and 
number of templates. 

tie extra implementation eost and ean easily be 
eombined with existing state-of-the-art systems. 
Future work will remove the fixed ordering for 
feature templates, and dynamieally add additional 
features based on the eurrent seores of different la¬ 
bels. 
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Model 

Templates 

Accuracy 

Speed 

Baseline 

46 

97.22 

lx 

Lasso 

6 

90.53 

10.97X 

Lasso 

10 

94.33 

7.67x 

Lasso 

15 

94.84 

5.23x 

Lasso 

23 

96.19 

3.31x 

GOMP 

6 

92.18 

6.83x 

GOMP 

10 

94.15 

4.29x 

GOMP 

15 

96.46 

2.83x 

GOMP 

23 

96.81 

1.96x 

Wrapper 

6 

96.45 

6.13x 

Wrapper 

10 

96.88 

3.82x 

Wrapper 

15 

97.01 

2.62x 

Wrapper 

23 

97.15 

1.70x 


Table 4: Comparison of the wrapper method for learning template orderings with group lasso and group 
orthogonal matehing pursuit. Speeds are measured relative to the baseline, whieh is about 23,000 to- 
kens/seeond on a 2.1GHz AMD Opteron maehine. 


Supplementary Material 

A Experiments: Learning Template Ordering 

As deseribed in Seetion 4.5, we experimented with 3 different algorithms for learning an ordering of 
feature templates: Group lasso (Lasso), whieh prunes feature templates based on their norm after £1 reg¬ 
ularization; Group orthogonal matehing pursuit (GOMP), whieh seleets features by iteratively training a 
model using the existing features then adds the template that maximizes Eqn. 4; and the wrapper method 
(wrapper). We use the methods of setting regularization parameters for Lasso and GOMP diseussed in 
Seetion 4.5. Note that for the ease of Group Lasso, this eorresponds to the experimental setup used when 
evaluating Group Lasso for NLP in Martins et al. (2011). 

We eompare the different methods for pieking template orderings in Table 4, training and testing 
models for WSJ POS tagging. To keep eoneerns separate, we use standard sequential predietion using 
all templates rather than our dynamie predietion method. We found the wrapper method to be the most 
sueeessful towards our goal of aehieving high aeeuraey while using as few templates as possible, whieh 
is important when using dynamie predietion. While the wrapper method eomes within 0.15% of state-of- 
the-art aeeuraey using the first 23 out of 46 total templates, neither GOMP nor Lasso are able to exeeed 
97% aeeuraey with so few templates. 

In terms of speed. Lasso outperforms both GOMP and the wrapper method, predieting 3 times as fast 
with 23 templates as the baseline, whereas GOMP prediets twiee as fast, and the wrapper method 1.7 
times baseline speed. It is initially puzzling that for a given number of templates, eaeh model does not 
aehieve the same speed inerease. Sinee eaeh template has only a single aetive feature for a given test 
instanee, we are eomputing the same number of multiplieations and additions for eaeh model. However, 
the different models are seleeting very different feature templates, whose features ean take a widely 
different amount of time to eompute. For example, ereating and hashing the string representing a trigram 
eonjunetion into a sparse veetor takes mueh longer than ereating a feature whose template has only two 
possible values, true wA false. 

This behavior is a refleetion of some interesting properties of the template seleetion methods, whieh 
help to explain the superior performanee of the wrapper method. 

Sinee the Group Lasso penalizes the norms of the templates as a surrogate for the sparsity (a 0-1 loss), 
it is naturally biased against large templates and will always inelude very small templates - the early 




stages of the regularization path will contain these small templates whose features can be quickly com¬ 
puted. Another likely source of speedup arises from the impact on cache locality of using much smaller 
cardinality weight vectors. This also explains the poor performance of the induced template ordering. In¬ 
cluding small templates early on runs at cross-purposes to our goal of placing highly predictive templates 
up front for our dynamic prediction algorithm. 

In contrast, the wrapper method and GOMP both pick larger, more predictive templates early on since 
they attempt to minimize loss functions that are unrelated to template size. While GOMP produces 
better orderings than lasso, in this case the wrapper method works better because the template ordering 
produced by GOMP is unable to incorporate signal from a validation set and overfits the training set. 

We find fhaf fhe wrapper mefhod works well wifh our dynamic fraining objective, which benefifs from 
predicfive, fhough expensive, fealures early on in fhe ordering. When using dynamic predicfion, fhe 
speed per femplafe used is offsel by using significanfly fewer femplafes on examples fhaf are easier fo 
classify. 

B Sparse Regularized Group Orthogonal Matching Pursuit 

Group Orfhogonal Mafching Pursuif (GOMP) picks a sfagewise ordering of fealure templates to add to 
a generalized linear model. At each stage, GOMP effectively uses each feature template to perform a 
linear regression to fit the gradient of the loss function. This attempts to find the correlation of each 
feature subset with the residual of the model. It then adds the feature template that best fits this gradient, 
and retrains the model. We adapt this algorithm to the setting of high-dimensional NLP problems by 
efficiently inverting the covariance matrices of the feature templates, and regularizing the computation 
of the residual correlation. This results in a scalable feature selection technique for our problem setting, 
detailed below. 

For purposes of exposition, we will break from the notation of Section 3 that combines features of x 
and y since the algorithm is designed from a linear-algebraic, regression standpoint that considers the 
design matrix X and the label matrix Y as separate entities. We will call each group of features Gi, its 
associated design matrix Xi (the feature matrix for that template). 

At each step k, we use our selected set of feature templates/groups and compute the gradient of our 
loss function on the training data set, call it and then we select a new feature template Gi with 
corresponding design matrix Xi to add to our selected groups, by finding the index that maximizes: 

arg Xi{Xj Xi)-^Xj (3) 

i 

After adding this template, we retrain the model on the selected set of templates and repeat. Note that this 
appears to be a difficult optimization problem to solve: some of our templates have hundreds of thousands 
or even millions of features and computing the inverse {XjXi)~^ could be expensive - however, due to 
the special structure of our NLP problems, where each feature template contains one-hot features, this 
covariance matrix is diagonal and hence trivially invertible. 

However, we find in practice that due to the large number of features and relatively small number of 
examples in our NLP models, this picks very-high cardinality feature templates early on that generalize 
poorly. The reason becomes apparent when we notice that the correlation-finding subroutine. Equation 
(3), is essentially an un-regularized least squares problem, attempting to regress the loss function gradient 
onto the data matrices for each template. This suggests we should try a form of regularization, by using 
some template-dependent constant a* to regularize the inversion of the covariance matrix: 

aTgmaxtr{{A^Y (4) 

i 

Heuristically, we pick this regularization parameter to be a low fractional power of the dimension size 
for each feature template A* = col(Ai)"% where ai is picked to be in the regime of [0.25, 0.5]. 



